Feature/add mlx #73

LeonNissen · 2024-10-16T18:28:23Z

Replace Llama.cpp with MLX

♻️ Current situation & Problem

Llama.cpp has been a reliable, multiplatform option. However, Apple’s recent release of MLX provides significant performance improvements, especially optimized for Apple Silicon devices. This PR proposes migrating to MLX to leverage these optimizations.

⚙️ Release Notes

Note: MLX-Swift is currently incompatible with simulators, so testing and deployment require a physical Apple Silicon device. Additionally, when using MLX with large language models (LLMs), the Increase Memory Limit Entitlement must be added to the application’s entitlements for optimal functionality.

This update removes certain APIs from the existing configuration. Please ensure your implementation aligns with the new MLX API.

🚀 Benchmark

Benchmarks indicate substantial performance gains across all tested devices. Refer to the figure below for detailed results:

📝 Code of Conduct & Contributing Guidelines

By submitting creating this pull request, you agree to follow our Code of Conduct and Contributing Guidelines:

I agree to follow the Code of Conduct and Contributing Guidelines.

PSchmiedmayer

@LeonNissen Thank you for the nice additions and working on this PR; very important and great to move to MLX for this package!

I have added some higher-level comments that I identified and added to the git diff.

Please ensure that all UI tests are passing + REUSE and SwiftLint elements are fixed. I also observe a lot of Swift 6 warning in the PR. I know that @paulhdk is also having an open PR where he is addressing most of them; please ensure that you sync with him and @philippzagar to ensure that these elements are in sync.

Apart from this I am happy to see this merged and addressed in this PR; we might want to tag a 2.0 beta version after this is merged and we had some time to bake it a bit together with the PRs from @paulhdk 🚀

Package.swift

Sources/SpeziLLMLocal/Helpers/LLMModel+numParameters.swift

Sources/SpeziLLMLocal/Helpers/ModelConfiguration+PromptFormat.swift

Sources/SpeziLLMLocal/LLMLocalPlatform.swift

Sources/SpeziLLMLocal/LLMLocalSession+Generate.swift

Sources/SpeziLLMLocal/LLMLocalSession+Setup.swift

Sources/SpeziLLMLocal/Resources/Localizable.xcstrings

Sources/SpeziLLMLocalDownload/LLMLocalDownloadManager.swift

Sources/SpeziLLMLocalDownload/LLMLocalLoadingManager.swift

PSchmiedmayer

Thank you for the updates; let me know once you need an additional round of reviews and have identified a way to pass all UI tests and remaining checks 🚀

Sources/SpeziLLMLocal/LLMLocalPlatform.swift

add note to README

fix memory selection

PSchmiedmayer

Thank you @LeonNissen; amazing additions and thank you for incorporating all the feedback!

Sources/SpeziLLMLocal/LLMLocalPlatform.swift

codecov · 2024-10-29T14:29:12Z

Codecov Report

Attention: Patch coverage is 0% with 269 lines in your changes missing coverage. Please review.

Project coverage is 36.24%. Comparing base (6633d8a) to head (a251d8a).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
...urces/SpeziLLMLocal/LLMLocalSession+Generate.swift	0.00%	93 Missing ⚠️
...peziLLMLocalDownload/LLMLocalDownloadManager.swift	0.00%	46 Missing ⚠️
...es/SpeziLLMLocal/Configuration/LLMLocalModel.swift	0.00%	38 Missing ⚠️
Sources/SpeziLLMLocal/LLMLocalSession+Setup.swift	0.00%	34 Missing ⚠️
Sources/SpeziLLMLocal/LLMLocalSession.swift	0.00%	19 Missing ⚠️
...SpeziLLMLocal/Helpers/LLMModel+numParameters.swift	0.00%	16 Missing ⚠️
Sources/SpeziLLMLocal/LLMLocalPlatform.swift	0.00%	8 Missing ⚠️
.../Configuration/LLMLocalPlatformConfiguration.swift	0.00%	5 Missing ⚠️
...s/SpeziLLMLocalDownload/LLMLocalDownloadView.swift	0.00%	5 Missing ⚠️
...eziLLMLocal/Configuration/LLMLocalParameters.swift	0.00%	2 Missing ⚠️
... and 2 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #73      +/-   ##
==========================================
+ Coverage   30.88%   36.24%   +5.36%     
==========================================
  Files          67       64       -3     
  Lines        3002     2503     -499     
==========================================
- Hits          927      907      -20     
+ Misses       2075     1596     -479

Files with missing lines	Coverage Δ
...ocal/Configuration/LLMLocalContextParameters.swift	`0.00% <ø> (ø)`
Sources/SpeziLLMLocal/LLMLocalError.swift	`0.00% <ø> (ø)`
...cal/Configuration/LLMLocalSamplingParameters.swift	`0.00% <0.00%> (ø)`
...eziLLMLocal/Configuration/LLMLocalParameters.swift	`0.00% <0.00%> (ø)`
Sources/SpeziLLMLocal/LLMLocalSchema.swift	`0.00% <0.00%> (ø)`
.../Configuration/LLMLocalPlatformConfiguration.swift	`0.00% <0.00%> (-100.00%)`	⬇️
...s/SpeziLLMLocalDownload/LLMLocalDownloadView.swift	`0.00% <0.00%> (ø)`
Sources/SpeziLLMLocal/LLMLocalPlatform.swift	`0.00% <0.00%> (-40.62%)`	⬇️
...SpeziLLMLocal/Helpers/LLMModel+numParameters.swift	`0.00% <0.00%> (ø)`
Sources/SpeziLLMLocal/LLMLocalSession.swift	`0.00% <0.00%> (ø)`
... and 4 more

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6633d8a...a251d8a. Read the comment docs.

Leon Nissen added 2 commits October 15, 2024 18:52

intermediate commit

3ba2134

add simulator check

519dac1

PSchmiedmayer requested changes Oct 16, 2024

View reviewed changes

PSchmiedmayer assigned LeonNissen Oct 16, 2024

PSchmiedmayer added the enhancement New feature or request label Oct 16, 2024

Leon Nissen added 2 commits October 17, 2024 19:04

improve code to PR comments

85df6f1

fix REUSE

d74b647

PSchmiedmayer reviewed Oct 21, 2024

View reviewed changes

Sources/SpeziLLMLocal/LLMLocalPlatform.swift Show resolved Hide resolved

Leon Nissen added 9 commits October 22, 2024 13:31

adjust UITest project

03fc955

add note to README

update readme

accb8ef

intermediate commit

b2ceaa5

fix liniting issues

4260756

add comments

e99b35d

fix memory selection

intermediate commit

fded07f

remote swiftlint

4480b8c

intermediate commit

768cbfc

skip test on release ipad

3ea48b0

LeonNissen changed the base branch from integration/mlx to main October 29, 2024 02:01

PSchmiedmayer approved these changes Oct 29, 2024

View reviewed changes

Sources/SpeziLLMLocal/LLMLocalPlatform.swift Show resolved Hide resolved

skip openai test due to same issue

a251d8a

PSchmiedmayer merged commit a4ee2da into main Oct 29, 2024
17 of 18 checks passed

PSchmiedmayer deleted the feature/add-MLX branch October 29, 2024 14:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/add mlx #73

Feature/add mlx #73

LeonNissen commented Oct 16, 2024 •

edited

Loading

PSchmiedmayer left a comment

PSchmiedmayer left a comment

PSchmiedmayer left a comment

codecov bot commented Oct 29, 2024 •

edited

Loading

Feature/add mlx #73

Feature/add mlx #73

Conversation

LeonNissen commented Oct 16, 2024 • edited Loading

Replace Llama.cpp with MLX

♻️ Current situation & Problem

⚙️ Release Notes

🚀 Benchmark

📝 Code of Conduct & Contributing Guidelines

PSchmiedmayer left a comment

Choose a reason for hiding this comment

PSchmiedmayer left a comment

Choose a reason for hiding this comment

PSchmiedmayer left a comment

Choose a reason for hiding this comment

codecov bot commented Oct 29, 2024 • edited Loading

Codecov Report

LeonNissen commented Oct 16, 2024 •

edited

Loading

codecov bot commented Oct 29, 2024 •

edited

Loading